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Executive  Summary 


This  root  cause  analysis  of  the  cost  and  schedule  overruns  of  the  Expeditionary 
Combat  Support  System  (ECSS)  was  sponsored  by  the  Director  of  Performance 
Assessment  and  Root  Cause  Analysis  (D,PARCA)  within  the  Office  of  the  Secretary  of 
Defense  (OSD).  This  analysis  is  one  of  three  perfonned  for  D,PARCA  at  the  request  of 
senior  officials  within  OSD  to  understand  the  DoD’s  large  Enterprise  Resource  Planning 
(ERP)  programs.  IDA  has  examined  ECSS  and  Global  Combat  Support  System  Marine 
Corps  (GCSS-MC),  while  the  RAND  Corporation  studied  Navy  ERP. 

ECSS  is  an  ERP  that  the  Air  Force  is  acquiring  to  transfonn  logistics.  The  31 
October  2010  Major  Automated  Information  System  (MAIS)  Quarterly  Report  (MQR) 
detennined  that  the  ECSS  program  had  incurred  a  “critical  change”  because: 

10  U.S.C.  Ch  144A  as  amended  by  the  FY2010  National  Defense 
Authorization  Act  (NDAA)  requires  Major  Automated  Information 
System  (MAIS)  programs  to  achieve  a  Full  Deployment  Decision  (FDD) 
within  five  years  after  funds  were  first  obligated  for  the  program.  Funds 
first  obligated  for  Increment  1  of  the  ECSS  program  occurred  on  31  Aug 
05  when  the  Milestone  Decision  Authority  (MDA)  approved  Milestone  A. 

As  of  3 1  Aug  10,  the  MDA  had  not  approved  an  Increment  1  FDD  thereby 
meeting  the  definition  of  a  critical  change. 

This  report  looks  at  the  root  causes  of  the  problems  in  the  ECSS  program  and  is  not 
restricted  to  the  critical  change.  While  the  program  does  not  have  an  Approved  Program 
Baseline  (APB),  and  therefore  has  no  official  estimate  of  cost  growth,  the  Government 
Accountability  Office  (GAO)  reported  that  its  estimated  cost  grew  from  $3.0  billion  in 
2008  to  $5.2  billion  as  of  their  report  in  October  2010. 2  Of  similar  importance,  the  initial 
plan  called  for  FDD  by  2010;  it  now  appears  that  the  final  FDD  (the  original  plan  only 
had  one)  will  not  occur  until  at  least  2016.  Therefore,  a  six-year  schedule  slip  has 
occurred. 


A  critical  change  on  a  MAIS  program  is  automatically  triggered  by  certain  conditions  relating  to  cost  or 
schedule.  A  critical  change  leads  to  a  formal  review,  which  then  typically  leads  to  changes  in  how  the 
program  is  executed. 

Government  Accountability  Office,  Report  No.  GAO-1 1-53  (October  2010). 


Methodology 

We  visited  with  ECSS’  sponsor  at  the  Air  Staff,  the  program  manager  (PM),  his 
staff,  and  the  contractor's  staff  to  understand  their  positions.  At  our  meeting  with  him,  the 
PM  gave  us  large  briefing  books  that  contain  a  great  deal  of  infonnation  about  the 
program  that  we  found  nowhere  else.  We  also  studied  all  publicly  available  data, 
including  quarterly  reports  to  the  Congress  and  the  Air  Force  budget  justification  books. 
Lastly,  we  read  ERP  literature  and  consulted  with  ERP  experts,  both  inside  and  outside 
the  government.  This  report  is  a  synthesis  of  what  we  learned  from  those  sources. 

Cost  Growth  and  Schedule  Slip 

The  GAO  asserts  that  the  costs  grew  75  percent  from  Milestone  (MS)  A  to  today 
(not  yet  MS  B),  while  the  program  manager  counters  that  34  percent  is  the  proper  number 
because  the  MS  A  estimate  and  the  latest  estimate  assume  different  levels  of  risk.  We 
have  nothing  with  which  to  compare  either  number,  to  be  able  to  determine  whether  or 
not  the  number  is  large.  The  standard  Nunn-McCurdy  thresholds  do  not  apply  in  this  case 
because  they  only  refer  to  an  APB,  which  the  MS  A  service  cost  position  (SCP)  is  not. 

The  PM  claimed  that  the  largest  portion  of  the  growth  in  the  cost  estimates,  $918 
million,  was  risk  that  was  not  accounted  for  at  MS  A.  We  do  not  know  precisely  what  he 
meant  by  “risk”  in  this  context.  In  addition,  he  listed  four  categories  of  cost  growth:  data 
readiness  ($544  million),  deliberate  delivery  ($345  million),  requirements  gaps  ($271 
million),  and  schedule  delays  ($166  million). 

The  Air  Force’s  2007  budget  listed  only  one  FDD  for  this  program  in  2010,  while  as 
of  January  2011  the  final  of  four  FDDs  now  specified  for  the  program  is  not  planned  to 
occur  until  at  least  2016,  a  slip  of  six  years.  Using  the  program  office’s  (PO)  assertions 
about  the  financial  benefits  of  using  ECSS,  we  estimate  that  this  accounts  for  about  $8 
billion  in  lost  benefits,  though  this  number  may  be  high. 

Root  Cause 

None  of  the  root  causes  of  the  cost  growth  in  the  ECSS  program  came  from 
unpredictable  exogenous  events  since  MS  A.  The  deepest  root  cause  of  cost  growth  and 
schedule  delay  in  ECSS  came  about  because  the  people  who  began  this  program  had 
insufficient  expertise  in  what  they  were  buying.  As  the  scope  of  ECSS  is  so  big,  it  is 
possible  that  nobody  in  the  world  really  knew  how  long  it  would  take  or  what  it  would 
cost  to  acquire  this  system.  It  may  also  be  that  it  is  impossible  to  know  how  much  a  large 
ERP  will  cost  until  the  blueprinting — a  significant  portion  of  the  cost — is  done. 

Of  the  standard  categories  for  cost  growth  that  the  Weapons  System  Acquisition 
Reform  Act  of  2009  lays  out,  this  program  clearly  suffered  from  one  “unanticipated 
design,  engineering,  manufacturing,  or  technology  integration  issue[s]  arising  during 
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program  performance”  (the  data  importation  issues).  There  is  also  evidence  that 
“government  or  contractor  personnel  responsible  for  program  management”  did  not  fully 
report  shortfalls  in  program  accomplishments.  Lastly,  the  program  was  launched  with  an 
insufficient  understanding  of  the  problems  involved  and,  consequently,  had  an 
unrealistically  low  cost  estimate. 
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1.  Background 


This  root  cause  analysis  of  the  cost  and  schedule  overruns  by  the  Expeditionary 
Combat  Support  System  (ECSS)  was  sponsored  by  the  Director  of  Performance 
Assessment  and  Root  Cause  Analysis  (D,PARCA)  within  the  Office  of  the  Secretary  of 
Defense  (OSD).  This  analysis  is  one  of  three  performed  for  D,PARCA  at  the  request  of 
senior  officials  within  OSD  to  understand  the  Department  of  Defense’s  (DoD)  large 
Enterprise  Resource  Planning  (ERP)  programs.  IDA  has  examined  ECSS  and  Global 
Combat  Support  System  Marine  Corps  (GCSS-MC),  while  the  RAND  Corporation 
studied  Navy  ERP. 

ECSS  is  an  ERP  that  the  Air  Force  is  acquiring  to  transfonn  logistics.  The  31 
October  2010  Major  Automated  Information  System  (MAIS)  Quarterly  Report  (MQR) 
detennined  that  the  ECSS  program  had  incurred  a  “critical  change”1  because: 

10  U.S.C.  Ch  144A  as  amended  by  the  FY  2010  National  Defense 
Authorization  Act  (NDAA)  requires  Major  Automated  Information 
System  (MAIS)  programs  to  achieve  a  Full  Deployment  Decision  (FDD) 
within  five  years  after  funds  were  first  obligated  for  the  program.  Funds 
first  obligated  for  Increment  1  of  the  ECSS  program  occurred  on  31  Aug 
05  when  the  Milestone  Decision  Authority  (MDA)  approved  Milestone  A. 

As  of  3 1  Aug  10,  the  MDA  had  not  approved  an  Increment  1  FDD  thereby 
meeting  the  definition  of  a  critical  change. 

This  report  looks  at  the  root  causes  of  the  problems  in  the  ECSS  program  and  is  not 
restricted  to  the  critical  change.  While  the  program  does  not  have  an  Approved  Program 
Baseline  (APB),  and  therefore  has  no  official  estimate  of  cost  growth,  the  Government 
Accountability  Office  (GAO)  reported  that  its  estimated  cost  grew  from  $3.0  billion  in 
2008  to  $5.2  billion  as  of  their  report  in  October  2010. 2  Of  similar  importance,  the  initial 
plan  called  for  FDD  by  2010;  it  now  appears  that  the  final  FDD  (the  original  plan  only 
had  one)  will  not  occur  until  at  least  2016.  Therefore,  a  six-year  schedule  slip  has 
occurred. 


A  critical  change  on  a  MAIS  program  is  automatically  triggered  by  certain  conditions  relating  to  cost  or 
schedule.  A  critical  change  leads  to  a  formal  review,  which  then  typically  leads  to  changes  in  how  the 
program  is  executed. 

Government  Accountability  Office,  Report  No.  GAO-1 1-53  (October  2010). 
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A.  ERPs 

An  ERP  is  an  Infonnation  Technology  (IT)  system  used  to  run  parts  of  the  business 
operations  of  an  enterprise.  An  ERP  is  typically  composed  of  a  set  of  subsystems,  each 
handling  a  set  of  key  processes,  which  could  include  an  organization’s  human  resources 
management,  financial  management,  logistics,  or  administrative  services.  An 
organization  could  select  to  implement  only  one  subsystem  or  a  combination  of 
subsystems  depending  on  business  need.  The  ERP  is  designed  to  optimize  an 
organization’s  business  processes  by  creating  a  central  repository  for  all  business  data  to 
flow  through.  The  goal  of  the  ERP  is  to  supply  employees  with  the  most  recent  and 
relevant  information,  whether  it  be  personnel-related  data,  the  availability  of  a  part  in 
inventory,  the  location  of  a  part  in  transit,  or  the  payment  for  said  part,  among  many  other 
types  of  data. 

1.  Purpose  of  an  ERP 

ERP  advocates  explain  that  the  primary  advantages  of  having  an  ERP  are  visibility 
and  efficiency.  An  ERP  has  at  its  core  a  single  relational  database  that  tracks  everything 
within  its  purview.  Visibility  means  that  this  database  allows  users  to  see  the  enterprise  as 
a  whole,  which  can  be  useful  for  both  managers  at  the  top  who  want  to  see  what  their 
enterprise  is  doing  and  those  in  the  trenches  who  need  infonnation  to  do  their  jobs. 

Efficiency  comes  through  the  use  of  common  best  practices  across  the  enterprise.  If 
an  ERP  is  implemented  across  the  Air  Force  for  the  processes  of  maintaining  trucks, 
mechanics  at  Eglin  Air  Force  Base  (AFB)  in  Florida  will  use  the  same  processes  as 
mechanics  at  Elmendorf  AFB  in  Alaska;  if  these  processes  are  well  designed,  the 
enterprise  as  a  whole  will  benefit.  While  the  trucks  at  both  bases  may  have  been  identical 
when  they  were  new,  the  problems  they  develop  over  their  lives  are  likely  to  be  different. 
For  example,  road  salt  may  cause  certain  parts  to  conode  quickly  in  Alaska,  while  those 
parts  may  last  much  longer  in  Florida.  Designing  processes  that  are  optimal  in  both 
circumstances  is  difficult. 

2.  Commercial  ERP  Software 

There  are  two  large  commercial  suppliers  of  ERP  software:  Oracle  and  SAP.  The 
ERP  software  they  license  is  considered  a  commercial-off-the-shelf  (COTS)  item.  The 
COTS  software  is  essentially  a  toolkit  for  creating  an  ERP;  any  ERP  implementation 
requires  a  significant  amount  of  configuration  and  customization  of  the  software. 
Business  processes  either  need  to  be  modified  to  confonn  to  the  constraints  of  the  COTS 
software  or  the  software  needs  to  be  modified  to  match  the  processes  by  adding  reports, 
interfaces,  conversions,  or  extensions  (collectively  called  RICE  objects). 
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3.  Implementing  an  ERP 

The  process  of  installing  an  ERP  is  often  referred  to  as  implementing  it  and  is 
carried  out  by  specialists  called  system  integrators  (SI).  In  many  cases,  organizations  that 
want  to  implement  an  ERP  hire  a  firm  that  specializes  in  ERP  implementation  to  do  the 
work.  This  is  what  DoD  has  done  in  both  of  the  cases  we  studied. 

The  process  of  implementing  an  ERP  requires  several  steps.  First,  the  SI  team 
blueprints  the  ERP.  This  means  that  they  look  at  what  the  enterprise  does  and  what  the 
client  wants  the  ERP  to  do.  For  example,  the  Air  Force  maintains  C-17  cargo  aircraft  and 
plans  where  they  will  go.  ECSS  will  handle  all  maintenance  of  the  C-17s,  but  it  will  not 
schedule  where  they  fly.  Handling  C-17  maintenance  means  that  the  ERP  will  need  to 
track  all  of  the  parts  and  supplies  used  to  maintain  the  aircraft  and  know  the  processes 
used.  In  the  blueprinting  stage,  the  SI  will  record  what  these  supplies  and  processes  are 
and  how  they  tie  to  other  processes.  The  C-17  maintainers  will  need  tires,  so  it  is  at  this 
stage  that  the  SI  team  will  look  at  the  processes  involved  in  storing  and  installing  tires,  as 
well  as  ordering  new  tires  and  disposing  of  old  ones. 

After  blueprinting,  the  SI  must  perfonn  a  “fit-gap  analysis.”  This  analysis  maps  the 
processes  that  the  client  currently  uses  into  the  ERP  and  identifies  any  gaps.  The  SI  team 
meets  with  the  client’s  practitioners  to  leam  about  their  processes  and  how  well  they 
match  with  processes  built  into  the  ERP.  Some  organizations  may  take  this  opportunity  to 
streamline  and  optimize  their  business  processes  to  ensure  consistency  and  efficiency. 
Both  groups  need  to  come  to  a  conclusion  as  to  how  the  processes  will  be  handled  once 
the  ERP  is  complete.  For  example,  the  COTS  software  would  have  a  process,  possibly 
with  multiple  options,  for  receiving  goods  at  a  warehouse.  If  this  process  is  the  same  as 
the  one  that  the  client  has  been  using,  it  will  be  implemented  as  is.  However,  if  the 
process  does  not  match  current  practice,  either  the  practitioners  must  change  their  process 
to  match  the  COTS  software  or  the  SI  team  will  have  to  create  RICE  objects  to 
accommodate  the  client’s  requirements.  This  process  is  often  long  and  sometimes 
contentious.  The  “fit”  is  where  the  COTS  software  matches  organizational  need  and  the 
“gaps”  are  where  RICE  objects  will  be  needed. 

Another  phase  of  ERP  implementation  is  populating  the  database.  That  is  typically 
perfonned  towards  the  end;  however,  it  must  be  considered  and  planned  for  from  the 
beginning  in  order  to  be  successful.  This  can  involve  perfonning  physical  inventories  of 
what  the  enterprise  has  as  well  as  converting  data  and  importing  it  from  legacy  IT 
systems.  Data  conversion  is  often  labor  intensive,  especially  when  there  are  many  legacy 
IT  systems  that  store  data  in  different  ways.  Legacy  data  must  first  be  mapped  to  the  new 
ERP  and  then  cleansed  to  ensure  data  compatibility.  Once  data  conversion  is  complete, 
data  validation  must  be  perfonned  by  knowledgeable  staff  to  ensure  accuracy  and 
completeness.  If  an  ERP  is  going  to  be  used  for  forecasting,  as  ECSS  is  expecting  in  its 
second  release,  then  in  addition  to  populating  the  database  with  the  current  status  of  the 
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enterprise,  the  client  will  need  to  load  historical  data.  Converting  historical  data  is  often 
more  challenging  than  working  with  current  data. 

Although  we  present  these  steps  as  though  they  are  a  procedure,  they  are  generally 
all  part  of  an  iterative  process.  ERPs  are  usually  brought  online  gradually  and  are  works 
in  progress  for  some  time.  Often  in  the  early  stages,  there  are  pilots  in  which  one  part  of 
the  enterprise  will  test  out  a  part  of  the  ERP,  followed  by  the  addition  of  more  processes 
and  more  users  until  a  full  release  across  the  enterprise  becomes  feasible.  Releases  are 
also  often  rolled  out  by  location  before  the  entire  enterprise  is  involved.  For  example,  the 
Air  Force  does  not  expect  that  Release  1  (Rl)  of  ECSS  will  begin  running  everywhere 
simultaneously.  The  Air  Force  will  introduce  it  at  a  few  locations  at  a  time  so  that  the  SI 
team  can  train  users  in  one  location  and  then  move  on  to  train  new  people  in  other 
locations.  As  that  goes  on  they  will  likely  be  making  small  changes  as  well,  either  to 
improve  performance  or  comply  with  exogenous  changes  in  the  environment. 

B.  ECSS 

The  sponsor  for  ECSS  is  Mr.  Grover  Dunn,  the  Director  of  Transformation,  Deputy 
Chief  of  Staff  for  Logistics,  Installations  and  Mission  Support,  Headquarters  U.S.  Air 
Force.  He  told  us  that  his  goal  is  to  transform  the  way  logistics  is  done  in  the  Air  Force, 
and  he  sees  ECSS  as  an  enabling  tool.  Today,  there  are  more  than  200  IT  systems  being 
used  around  the  globe  by  Air  Force  personnel  to  conduct  logistics.  Transforming  so  many 
systems  individually  is  a  virtual  impossibility;  instead,  ECSS  is  intended  to  replace  them 
all. 

In  July  2009,  there  was  some  restructuring  of  the  program  office  (PO).  Kenneth 
Moran  was  promoted  to  brigadier  general  and  named  both  program  manager  (PM)  of 
ECSS  and  program  executive  officer  (PEO)  over  more  than  one  hundred  Air  Force 
logistics  IT  systems,  including  ECSS.  His  goal  is  to  have  ECSS  replace  all  of  the  other 
systems  in  his  portfolio  while  making  Air  Force  logistics  more  capable.  This 
management  structure  was  created  so  that  the  head  of  ECSS  would  have  the  necessary 
control  to  force  the  changes  required  to  make  the  ERP  implementation  succeed. 

Currently,  the  Milestone  Decision  Authority  (MDA)  for  ECSS  is  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L)). 

1.  Contractors 

The  first  major  contractor  hired  by  the  ECSS  PO  was  Oracle.  The  PO  selected 
Oracle  to  be  the  provider  for  the  COTS  software  in  their  ERP.  This  decision  was 
protested  by  SAP,  but  was  upheld  five  months  later  by  the  GAO,  and  the  PO  has  been 
licensing  Oracle’s  software  ever  since. 
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The  other  major  contractor  hired  was  Computer  Services  Corporation  (CSC).  CSC 
was  one  of  five  firms  that  were  allowed  to  compete  for  SI  contracts  under  the  Enterprise 
Software  Initiative  (ESI)  that  had  been  established  within  OSD.  Since  the  end  of  a  six- 
month  contract  protest  by  IBM — another  one  of  the  five  ESI-approved  contractors — CSC 
has  been  working  with  the  PO  to  create  ECSS. 

2.  Timeline 

Figure  1  shows  the  ECSS  timeline  as  presented  by  the  PM  in  January  2011.  ECSS 
had  problems  almost  from  the  start.  The  following  events  in  the  timeline  are  worth 
noting: 

•  In  November  2005,  three  months  after  MS  A,  SAP  protested  the  contract  award 
to  Oracle.  Then,  in  September  2006,  IBM  protested  the  SI  award  to  CSC. 

Though  both  protests  were  overruled,  they  caused  a  combined  downtime  for  the 
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program  of  1 1  months. 

•  In  May  2009,  ECSS  decided  to  switch  to  an  all-Oracle  product  suite.  Oracle’s 
bid  for  the  contract  in  2005  stated  that  ECSS  would  integrate  three  software 
packages  from  different  companies  to  create  their  ERP:  Oracle,  Click 
Commerce,  and  IFS.  This  was  deemed  necessary  because  Oracle’s  software  did 
not  have  sufficient  tools  to  run  a  heavy  maintenance  facility  or  for  long  term 
planning.  Integrating  those  functions  was  more  complicated,  and  consequently 
more  expensive,  than  the  PO  originally  expected.  By  2009,  the  Oracle  software 
had  progressed  enough  that  the  program  office  decided  not  to  use  Click  or  IFS, 
but  instead  to  build  the  ERP  entirely  from  Oracle  software.4  (The  PM  noted  in 
December  2010  that  an  Oracle-based  ERP  has  been  used  as  the  sole  IT  system  at 
an  aircraft  heavy  maintenance  facility  in  Europe.) 

•  The  last  thing  to  note  from  the  timeline  is  that  MS  B  does  not  appear.  It  has  been 
pushed  back  by  the  MDA  several  times  and  still  has  not  occurred  for  any  portion 
of  ECSS.5 


The  GAO  decisions  took  five  months  and  six  months,  respectively,  according  to  the  published  reports. 
Figure  1  is  from  the  PM’s  briefing  to  the  MDA  on  5  January  2011,  which  asserted  that  the  two 
combined  protests  cost  18  months. 

4 

ECSS  is  not  in  fact  an  all-Oracle  solution;  the  plan  calls  for  using  another  product  called  Ventureforth 
for  remote  access. 

5  The  oldest  ECSS  schedule  with  MS  B  on  it  that  we  found  was  in  the  FY  2008/2009  Air  Force  RDT&E 
budget  book,  volume  III.  It  showed  MS  B  scheduled  for  the  fourth  quarter  of  FY  2008. 
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Figure  1.  ECSS  Chronology 


3.  Release  Structure 

As  of  a  2009  Acquisition  Decision  Memorandum  (ADM),  ECSS  is  supposed  to 
have  four  releases  with  each  release  as  a  separate  acquisition  program.  All  cost  estimates 
given  to  PARCA  are  for  all  four  releases  combined.  While  MS  A  was  for  the  whole  of 
ECSS,  current  funding  and  the  first  MS  B  is  only  for  Rl.  Figure  2  shows  the  schedule 
and  plans  for  all  of  the  releases.  Rl  and  R4  are  supposed  to  be  released  broadly  across  the 
Air  Force,  while  R2  and  R3  are  only  going  to  be  used  at  a  small  number  of  locations.  R2 
will  implement  planning  and  management  functions  at  seven  headquarters  locations  and 
R3  will  be  fielded  at  six  heavy  maintenance  facilities. 

The  pilots  on  the  chart  are  instances  in  which  some  subset  of  the  release  in  question 
is  fielded  at  a  limited  number  of  locations.  These  pilots  help  find  bugs  and  improve 
performance  before  the  system  is  deployed  everywhere. 
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Version  1 0.2a  -  30  Nov  1 0 


Fourth  Release  and  Multiple  Pilots  Added  to  Reduce  Risk 


Figure  2.  ECSS  Program  Schedule  from  BG  Moran’s  5  January  2011  Briefing 


4.  Regulations  and  Reporting 

Over  its  history,  ECSS  has  qualified  as  both  a  MAIS  program  and  a  Major  Defense 
Acquisition  Program  (MDAP).  The  process  for  acquiring  each  has  undergone  some 
changes,  though  the  MDAP  process  has  been  the  more  stable  of  the  two  recently.  ECSS 
has  at  times  been  regulated  like  an  MDAP  and  at  other  times  like  a  MAIS 
program — today  it  is  considered  an  MDAP.  As  an  MDAP,  the  MDA  is  the  USD(AT&L); 
when  it  was  treated  as  a  MAIS  program,  the  MDA  was  the  Deputy  Chief  Management 
Officer  (DCMO). 

The  Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  system 
contains  very  little  data  on  ECSS.  We  were  given  six  MQRs,  but  they  do  not  contain 
much  infonnation.  They  do  not  contain  any  spending  plans  beyond  2013,  and  they  do  not 
report  any  information  on  dollars  spent  in  the  past.  Neither  the  term  “Oracle”  nor  “CSC” 
appears  in  any  of  the  MQRs.  The  MQRs  do  make  it  clear  that  ECSS  has  no  baseline  and 
they  have  not  submitted  an  initial  estimate  of  the  costs. 

There  is  also  no  earned  value  or  cost  reporting  data  for  any  of  the  contracts  in  this 
program.  This  is  discussed  below  in  the  section  on  COTS  acquisition  and  cost  reporting. 

5.  Status  Today 

A  November  2010  ADM  from  the  USD(AT&L)  postponed  MS  B,  which  had  been 
scheduled  for  early  FY  2011,  and  halted  work  on  everything  except  the  immediate  work 
on  Rl.  Pilot  A  was  run  at  Hanscom  AFB  and  Pilot  B  is  currently  running  there.  Finding 
infonnation  about  Pilot  A  was  difficult,  but  we  did  gain  access  to  an  informal  Analysis  of 
Alternatives  (AoA)  planning  document  from  OSD  dated  March  2011  that  says: 

Initial  capabilities  fielded  at  Hanscom  AFB  in  Release  1,  Pilot  A  that 
enable  vehicle  maintenance,  other  maintainable  items,  and  tools 
management  processes.  Lessons  learned  include  both  planned  (future 
pilot)  and  unplanned  (rework  or  future  release)  items:  product  data 
management  processes  must  be  enabled,  an  improved  tools  solution  is 
required,  reporting  capabilities  must  be  enabled,  and  financial  processing 
must  be  completed.  The  Pilot  as  it  stands  is  [n]ot  viable  for  fielding  to  the 
enterprise. 

This  indicates  that  there  is  more  work  to  do  even  on  this  limited  part  of  the  program,  and 
suggests  why  MS  B  has  continued  to  be  pushed  into  the  future. 

Figure  3  shows  the  actual  spending  from  the  Air  Force’s  budget  justification  books, 
along  with  the  projections.  The  triangular  data  points  show  next  year  projections,  the 
squares  show  current  year  projections,  and  the  filled  circles  show  budget  actuals.  In  no 
year  has  ECSS  spent  more  than  the  projected  budget,  so  cost  overruns  are  a  result  of 
work  being  added  in  the  future,  not  unexpected  costs  that  were  promptly  paid.  During  FY 
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2005,  the  Air  Force  asked  for  $212  million  in  FY  2007,  but  the  FY  2009  justification 
book  reports  that  ECSS  only  spent  $135  million  in  FY  2007.  It  is  possible  that  this 
funding  shortfall  caused  some  difficulty  for  the  program,  but  the  PM  did  not  mention  this 
as  an  issue.  Note  that  the  chart  is  not  cumulative. 


ECSS  Spending  from  USAF  Budget  Justification  Books 


Figure  3.  ECSS  Spending  from  Air  Force  Budget  Justification  Books 
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2.  Cost  Growth  and  Schedule  Slippage 


Formally,  ECSS  has  had  a  critical  change  because  of  schedule  slip.  The  formal  rules 
for  a  MAIS  program  call  for  “a  Full  Deployment  Decision  (FDD)  within  five  years  after 

o 

funds  are  first  obligated  for  the  program,”  but  ECSS  did  not  achieve  that.  The  resulting 
critical  change  led  to  techniques  designed  to  reduce  risk,  but  that  also  increased  cost. 

While  cost  growth  and  schedule  slippage  are  routinely  addressed  together  because 
they  are  inherently  linked,  we  have  decided  here  to  discuss  them  both  in  their  own 
sections. 

A.  Cost  Growth 

Figure  4  shows  the  areas  of  cost  growth  according  to  the  program  manager.  The 
quantities  in  this  chart  are  reported  in  then-year  (TY)  dollars.9  An  explanation  of  the 
numbers  follows  below. 


Note  that  when  the  critical  change  was  declared,  this  program  was  not  being  regulated  as  an  MDAP  as 
it  is  now,  but  rather  as  a  MAIS  program. 

Typically,  analysis  is  done  in  base  year  (BY)  dollars  instead  of  TY  dollars  to  account  for  inflation,  but 
in  this  study  all  quantities  are  reported  in  TY  dollars.  To  properly  convert  from  TY  to  BY  dollars 
requires  having  the  funding  laid  out  by  appropriation  and  year  by  year,  but  this  information  was  not 
provided  to  us.  The  data  that  split  the  dollars  by  appropriation  and  annual  separations  end  with  a  “To 
Complete”  entry,  which  contains  more  than  20  percent  of  the  total  dollars.  We  could  make  some 
educated  guesses  to  estimate  the  splits,  but  it  would  introduce  unnecessary  complication  into  an  analysis 
that  already  has  limited  precision. 
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Figure  4.  Cost  Increases  Combination  of  Cost  T ransfer  and  Growth  (1  of  2)  from  BG  Moran’s  5  January  2011  Briefing 


1.  Current  Estimate 

The  current  estimate  of  $5.2  billion  for  ECSS,  which  is  labeled  “MS  B”  in  Figure  4, 
is  not  straightforward  in  its  presentation.  The  PM  told  us  that  it  is  the  estimate  for  the 
entirety  of  ECSS — all  four  releases — and,  thus,  is  a  fair  number  to  compare  to  the  MS  A 
Service  Cost  Position  (SCP).  However,  the  MS  B  authority  that  the  program  office  is 
seeking  is  only  for  Rl.  OSD  and  the  PM  have  disagreed  on  whether  R1  is  mature  enough 
for  MS  B  authority,  which  suggests  that  this  portion  of  the  cost  estimate  may  not  be  very 
precise.  Since  both  agree  that  R2,  R3,  and  R4  are  not  yet  ready  for  MS  B,  there  is  even 
less  certainty  in  those  portions  of  the  cost  estimate.  We  do  not  have  the  necessary  data  to 
identify  the  Rl  costs  alone. 

2.  MS  A  Estimate  Disagreement 

Figure  4  shows  two  numbers  for  MS  A.  The  PM’s  claim  is  that  while  the  MS  A  SCP 
said  $3.0  billion,  that  estimate  did  not  have  the  same  risk  factor  as  the  current  $5.2  billion 
estimate.  To  make  the  estimates  equivalent,  the  PM  added  $0.9  billion  to  the  MS  A 
number.  Whether  or  not  we  accept  the  PM’s  claim  determines  whether  we  would  say  this 
program  experienced  34  percent  or  75  percent  cost  growth  since  MS  A.  The  PM  never 
asserted  what  the  MS  A  or  MS  B  confidence  levels  are  or  were. 

Whether  one  calls  the  cost  growth  34  percent  or  75  percent  from  MS  A  to  today — 
not  yet  at  MS  B — we  still  do  not  have  a  comparable  figure  in  order  to  determine  whether 
the  number  is  large.  The  standard  Nunn-McCurdy  thresholds  do  not  apply  in  this  case 
because  they  only  refer  to  an  APB,  which  the  MS  A  SCP  was  not. 

3.  PM’s  Cost  Growth  Categories 

The  PM  listed  four  categories  of  cost  growth,  which  are  discussed  below.  This  is  in 
addition  to  the  largest  category  of  growth — risk — which  was  discussed  immediately 
above. 

a.  Data  Readiness  -  PM  Estimate:  $544  Million 

Data  conversion  is  often  challenging  because  there  are  many  legacy  IT  systems,  and 
each  has  its  own  particular  database  structures,  data  fonnats,  and  data  definitions. 
Furthermore,  the  legacy  systems  may  have  inconsistencies  or  mistakes  in  them,  all  of 
which  need  to  be  painstakingly  corrected  by  both  the  people  who  know  the  data  and  the 
ERP  implementation  experts.  The  PO  knew  from  the  beginning  that  data  would  need  to 
be  imported  and  that  it  would  be  a  significant  expense. 

The  cost  growth  associated  with  data  readiness  has  two  components:  the  expected 
effort  required,  and  who  will  bear  the  costs.  When  the  ERP  team  started  looking  at  the 


13 


data  to  be  imported  into  the  ERP,  they  found  the  task  to  be  more  difficult  than  had  been 
anticipated.  We  have  no  evidence  that  this  cost  is  well  understood  today. 

Furthennore,  the  PM  reported  in  his  5  January  2011  briefing  that,  in  November 
2008,  the  program  office  absorbed  the  Data  Readiness  costs,  which  had  previously  been 
assigned  to  the  Air  Force’s  major  commands  (MAJCOMs).  This  second  point  is 
somewhat  in  contention.  The  program’s  sponsor  from  the  Air  Force  staff  told  us  in  an 
interview  that  it  had  always  been  his  intention  for  this  program  to  be  a  help — not  a 
hindrance — to  the  MAJCOMs,  and  he  had  decided  at  the  outset  to  bear  all  of  the  costs  in 
the  program,  including  data-related  costs. 

b.  Deliberate  Delivery  Strategy  -  PM  Estimate:  $345  Million 

The  PM  told  us  that  several  times  in  ECSS’  history  the  program  has  been  told  to 
reduce  risk,  either  by  the  GAO,  the  MDA,  or  by  parties  within  the  Air  Force.  Each  time, 
this  has  involved  doing  less  in  parallel  and  adding  more  steps,  such  as  pilots.  Each  of 
these  steps  reduces  the  risk  of  cost  growth  by  accepting  additional  cost  up  front.  The  PM 
said  that  most  of  these  steps  were  helpful  and  more  may  be  added  in  the  future.  However, 
because  the  early  cost  estimates  seemed  to  assume  ECSS  would  be  simpler  to  implement 
than  proved  to  be  the  case,  these  risk  mitigation  strategies — added  later — have  increased 
the  PO’s  cost  estimate. 

The  largest  such  mitigation  strategy  occurred  in  late  2009,  when  the  MDA  approved 
a  restructure  of  the  program.  ECSS  went  from  being  three  releases  within  a  single 
acquisition  program  to  four  releases,  each  of  which  is  its  own  separate  acquisition 
program.  Initially,  R1  would  have  been  all  of  the  parts  being  spread  around  the  entire  Air 
Force  with  R2  and  R3  being  smaller  releases  used  at  a  few  facilities  for  planning  and 
heavy  maintenance. 10  When  R4  was  created,  much  of  R1  was  moved  to  R4,  making  it  the 
largest  release  of  all  in  terms  of  fielding  locations.  R4  is  expected  to  include  flight  line 
maintenance,  which  is  particularly  challenging  because  it  requires  accessing  the  ERP  on 
portable  devices  that  personnel  can  use  next  to  an  aircraft  outdoors.  We  have  no 
particular  reason  to  believe  this  pattern  of  spending  more  to  alleviate  risk  is  over;  there 
may  be  more  risk  mitigation  costing  more  time  and  money  in  the  future  if  this  program  is 
granted  MS  B  authority. 

c.  Requirements  Increase  -  PM  Estimate:  $271  Million 

The  reported  cost  growth  from  increased  requirements  is  made  up  of  three  parts: 
Product  Lifecycle  Management  (PLM)  for  $132  million,  hardware11  for  $108  million  and 


It  is  unclear  if  the  original  R1  included  IFS  and  Click  Commerce  or  if  they  were  only  expected  to  be 
used  in  R2  and  R3 . 

This  entry  refers  to  IT  hardware,  such  as  computers,  cabling,  and  routers. 
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Logistics  Financials  (LogFins)  for  $31  million.  For  PLM,  the  claim  is  that  its  cost  was 
underestimated  at  MS  A  and  was  not  properly  accounted  for  in  the  initial  contract  with 
CSC.  The  current  cost  estimate  for  this  is  based  on  the  contract  negotiations  with  CSC 
that  were  approved  in  2009.  One  might  argue  that  this  is  not  a  change  in  requirements  but 
rather  a  flaw  in  the  early  cost  estimate.  The  PM  called  it  an  increase  in  requirements 
because  it  was  a  change  in  the  requirements  on  the  CSC  contract. 

The  hardware  cost  growth  comes  from  the  program  agreeing  to  buy  some  hardware 
for  the  MAJCOMs  to  run  ECSS.  Many  ECSS  users  will  access  the  ERP  using  computers 
they  already  have,  but  some  new  hardware  will  be  required  to  put  computers  in  places 
where  there  were  none  before  and  perhaps  to  strengthen  the  infrastructure  so  it  is  capable 
of  handling  the  increased  traffic  from  ECSS.  While  these  costs  are  not  new,  the  PM  told 
us  that,  at  MS  A,  hardware — like  data  readiness — would  be  paid  for  by  the  MAJCOMs, 
but  now  the  ECSS  PO  will  pay  for  it;  therefore,  it  is  a  new  requirement  for  ECSS. 

LogFins  was  originally  going  to  be  part  of  another  ERP  called  Defense  Enterprise 
Accounting  and  Management  System  (DEAMS).  That  ERP  is  sponsored  by  the  Air 
Force’s  finance  community,  and  was  originally  intended  to  track  all  Air  Force  financial 
transactions.  In  CSC’s  design  for  ECSS,  a  large  amount  of  data  transmission  would  have 
to  occur  between  ECSS  and  DEAMS  for  each  purchase,  requiring  the  creation  of  many 
RICE  objects.  However,  the  COTS  software  from  Oracle  could  handle  all  of  the  LogFins 
internally  relatively  easily,  even  in  2005.  By  moving  LogFins  into  ECSS,  the 
requirements  for  DEAMS  decreased,  while  ECSS’s  requirements  increased.  Because 
this  requires  less  interaction  between  the  two  systems,  this  change  should  decrease  the 
total  cost  to  the  Air  Force.  What  this  means  in  practice  is  that  ECSS  will  handle  all 
financial  transactions  for  the  Air  Force’s  maintenance  working  capital  fund,  in  addition  to 
its  originally  planned  capabilities. 

d.  Schedule  Delays  -  PM  Estimate:  $166  Million 

Schedule  delays  due  to  COTS  integration  and  contract  protest  are  estimated  by  the 
PM  to  have  cost  $159  million  and  $7  million,  respectively.  COTS  integration  refers  to  the 
original  attempt  to  use  Click  Commerce  and  IFS  software  along  with  Oracle’s  software  to 
build  ECSS.  The  ECSS  program  spent  time  and  money  to  integrate  those  packages  before 
the  non-Oracle  packages  were  ultimately  discarded  in  favor  of  an  all-Oracle  solution.  It  is 
our  opinion  that  changes  like  this  in  the  pre-MS  B  phases  of  a  program  should  be 
expected;  such  exploration  of  alternative  approaches  is  a  typical  part  of  the  requirements 
analysis  and  design  process,  and  cost  estimates  should  take  such  possibilities  into  account 
from  the  start. 


We  have  not  studied  DEAMS  and  therefore  cannot  say  if  their  cost  estimate  shows  a  decrease  from  this 
decision. 
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The  contract  protests  against  the  selections  of  Oracle  and  CSC  accounted  for  five 
and  six  months  of  delays  in  2006  and  2007,  respectively.  We  do  not  understand  why  such 
major  delays  in  a  program  with  this  sort  of  expenditure  rate  cost  only  $7  million.  ECSS 
spent  $60  million  in  2006  and  $135  million  in  2007. 

B.  Schedule  Slippage 

When  ECSS  (as  a  single  acquisition  program)  was  given  MS  A  authority,  there  was 
an  assumption  that  FDD  must  have  been  scheduled  no  later  than  FY  2010,  five  years 
from  program  inception.  The  schedules  provided  by  the  PM  do  not  say  when  the  four 
FDDs  will  be  (one  for  each  of  the  releases  in  the  current  plan),  but  they  do  show  an 
estimate  for  each  MS  C,  which  precedes  FDD.  According  to  Figure  2,  R1  FDD  will  occur 
after  January  2012,  which  is  two  years  late.  However,  since  all  of  ECSS  began  at  the 
same  time,  a  better  marker  for  the  schedule  slip  would  be  the  MS  C  for  R4,  which  is 
projected  to  be  in  May  2016,  six  years  late. 

1.  Causes  of  Schedule  Slip 

In  our  discussions  with  the  PM,  there  were  several  different  problems  that  caused 
the  schedule  to  change,  and  they  all  line  up  with  the  categories  for  cost  growth  discussed 
in  Section  2.A.2.  Most  elements  that  added  cost  added  work  to  the  program  and  caused 
the  final  delivery  date  to  move  further  into  the  future.  We  do  not  have  any  data  that  allow 
us  to  partition  the  schedule  changes  among  different  categories  similar  to  the  infonnation 
we  received  from  the  PM  on  cost  growth. 

2.  Consequences  of  Schedule  Slip 

The  primary  reason  for  purchasing  ECSS  is  that  it  will  allow  the  Air  Force  to  carry 
out  its  logistics  function  more  economically  than  by  continuing  to  use  all  of  the 
incumbent  systems,  thereby  providing  a  means  of  quantifying  the  benefits  of  fielding 
ECSS.  Delays  in  implementing  ECSS  effectively  mean  that  the  Air  Force  will  receive 
less  benefit  from  buying  it.  The  program  office  provided  detailed  estimates  of  the 
financial  benefits  of  ECSS  by  year  from  2011  through  2026,  and  it  is  reasonable  to 
consider  the  lost  benefit  as  being  similar  to  extra  cost. 14 


We  do  not  know  the  schedule  for  FDD  that  existed  at  MS  A.  If  it  was  scheduled  before  the  deadline,  the 
delays  are  even  longer  than  we  estimate. 

We  note  that  to  keep  this  analysis  consistent,  all  money  is  presented  as  non-discounted  TY  dollars.  If 
we  did  discount  these  dollars,  the  cost  from  deploying  ECSS  late  would  look  even  higher  because  the 
lost  savings  come  early  in  the  program’s  life.  However,  we  would  also  need  to  discount  the  costs  to  stay 
consistent  and  we  have  not  done  so  because  we  do  not  have  that  split  out  by  year,  as  noted  above. 
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It  is  notable  that  benefits  taper  off  after  2023;  that  occurs  because  each  release  of 
ECSS  is  assumed  to  have  a  limited  lifespan,  so  as  each  one  phases  out,  the  benefits  also 
end.  We  consider  that  this  is  an  artifact  of  the  assumptions  about  program  duration  and 
not  real.  To  estimate  the  annual  benefits  that  have  been  lost  from  six  years  of  schedule 
slippage,  we  take  the  values  of  the  typical  years,  2018  through  2023,  average  them,  and 
then  multiply  that  figure  by  six.  This  comes  to  about  $1.3  billion  per  year  for  six  years,  or 
a  total  of  $8  billion. 

Figure  5  shows  benefits  as  reported  by  the  PM.  Appendix  C  has  complete  data. 
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Benefits  of  ECSS  by  Year 


Source:  Data  from  ECSS  PM,  chart  by  IDA. 


Figure  5.  Benefits  of  ECSS  by  Year 


3.  Root  Causes  for  Cost  Growth 


In  Section  2,  we  detailed  the  specific  items  that  the  PM  believes  were  responsible 
for  the  cost  growth  in  ECSS  to  date.  While  we  generally  accept  the  PM’s  claims,  we 
think  there  are  root  causes  that  go  deeper. 

A.  DoD  ERPs  Are  Not  COTS  Acquisitions 

The  Oracle  and  SAP  software  packages  that  are  used  to  build  ERPs  are  COTS  items 
that  can  be  licensed  and  used  as  is.  However,  just  using  the  software  does  not  mean  that 
the  enterprise  has  an  ERP;  the  software  allows  an  ERP  to  be  built  and  run.  The  full  ERP 
has  a  populated  database  and  has  mapped  the  enterprise’s  procedures  to  the  software. 
Customizing  and  configuring  the  ERP  is  not  a  COTS  purchase. 15  This  relates  to  the 
perceived  change  in  risk  and  the  deliberate  delivery  strategy. 

1.  Modifications  to  the  COTS  Software 

In  the  world  of  ERP  implementations,  it  is  so  common  for  the  COTS  software  to  be 
insufficient  for  the  user  that  there  is  a  name  for  the  fixes:  RICE  objects.  It  is  neither 
surprising  nor  a  reason  for  concern  that  ECSS  includes  some  of  these — the  current 
estimate  is  250  in  Rl.16  However,  ECSS  requires  modifications  to  the  COTS  software 

i  n 

that  go  beyond  RICE  objects,  and  this  has  been  a  hallmark  of  other  DoD  ERPs  as  well. 

ECSS  has  at  least  three  requirements  that  go  beyond  what  Oracle’s  software  could 
do  at  MS  A:  mobile  connectivity,  heavy  maintenance,  and  planning.  For  the  mobile 
connectivity,  ECSS  will  use  a  product  from  Ventureforth,  Inc.  that  has  been  used  in  the 
past  with  Oracle  with  a  high  enough  success  rate  that  Ventureforth  has  been  designated 


Some  services  that  are  purchased,  like  home  moving,  might  be  considered  “customizable  COTS.”  This 
means  that  while  each  home  is  unique,  it  is  made  up  of  standard  parts,  and  moving  companies  know 
how  to  estimate  the  cost  and  schedule  for  any  typical  move.  ERP  implementation  in  some  instances 
may  be  like  this,  although  the  large  DoD  ERPs  like  ECSS  are  not. 

The  count  of  250  RICE  objects  in  Rl  is  a  dramatic  decrease  from  earlier  estimates.  However,  it  is 
unclear  how  much  of  that  reduction  in  number  is  because  they  have  been  moved  to  other  releases, 
especially  R4,  as  opposed  to  a  reduction  in  the  number  of  RICE  objects  needed  to  be  generated  in  total 
for  ECSS. 

Both  ECSS  and  the  Marine  Corps’  Global  Combat  Support  System  (GCSS-MC)  required  modifications 
to  the  Oracle  COTS  software.  In  both  cases,  Oracle  has  created  the  new  functionality  and  offered  it  for 
sale  to  commercial  customers. 


19 


1 8 

an  “Oracle  Certified  Partner.”  For  the  other  two  problems,  Oracle  now  has  code  that  is 
in  their  standard  software  suite. 

Oracle  has  agreed  to  make  changes  to  its  software  to  accommodate  ECSS  without 
charging  extra  for  the  work. 

2.  Non-COTS  Portion  of  Any  ERP  Implementation 

Even  the  simplest  ERP  is  more  complicated  than  typical  COTS  software — which  is 
why  an  SI  is  generally  hired  to  create  the  solution.  Common  examples  of  COTS  software 
are  Microsoft  Office  and  Mathematica;  the  software  is  provided  on  disks  or  downloaded 
to  the  user  and  then  installed  on  a  computer.  Once  the  installation  is  completed,  the 
software  is  ready  for  use.  At  most,  there  are  a  few  questions  during  installation  regarding 
such  things  as  optional  functions  or  the  desired  installation  location  on  your  hard  drive  or 
network.  The  purchaser  of  an  ERP  must  decide  what  processes  will  be  handled  by  the 
ERP  and  how  to  map  them  into  the  COTS  software  package.  For  an  ERP  to  be  useful,  it 
must  be  set  up  for  use  on  numerous  machines.  Old  data  must  be  imported  from  across  the 
enterprise.  If  the  ERP  is  going  to  be  used  for  forecasting,  it  must  have  historical  data  in 
addition  to  the  current  status  of  the  enterprise.  For  these  reasons  every  ERP — if  not  the 
software  used  to  make  it — is  unique,  not  COTS,  by  the  time  it  is  actually  implemented 
and  used. 

Another  reason  ERPs  are  not  COTS  is  that  they  involve  changing  the  way  people  do 
their  jobs.  For  example,  in  ECSS,  the  process  for  troubleshooting  a  check  engine  light  on 
a  truck  will  be  standardized  across  the  enterprise.  A  mechanic  may  not  be  able  to  order  a 
replacement  fan  belt  until  he  has  checked  the  oil  level,  because  the  checklist  puts  that 
first.  In  an  organization  like  the  Air  Force,  there  may  be  many  different  methods  of 
working  that  all  need  to  be  modified  to  match  the  software.  Furthermore,  the  procedures 
that  will  be  used  by  the  ERP  need  to  be  selected  so  that  everyone  can  use  them — the  goal 
being  to  improve  average  productivity.  These  processes  must  work,  and  ideally  be 
optimal,  in  the  Alaska  winter,  the  Florida  summer,  and  a  small  base  in  the  Afghan 
mountains  (recall  that  the  “E”  in  ECSS  stands  for  “Expeditionary”). 19 

Lastly,  ERPs  are  so  large  and  affect  so  many  different  people  and  processes  that  it  is 
exceedingly  difficult  to  know  from  the  outset  precisely  how  much  work  will  be  required. 
For  example,  the  requirements  for  a  new  helicopter  may  not  be  difficult  to  explain:  fly  at 
a  certain  speed,  carry  a  specific  weight  load,  have  a  designated  sensor  system,  and  so 
forth.  Yet  the  documents  for  these  straightforward  requirements  are  generally  a  few 


http://www.ventureforth.com/mobile  i2k.asp,  accessed  1  April  2011. 

In  a  discussion  with  an  ERP  expert  we  asked  if  it  were  possible  for  the  checklists  to  be  designed  so  that 
they  differ  by  season  or  location.  The  answer  is  in  theory  yes,  but  doing  so  would  rapidly  expand  the 
complexity  of  the  ERP. 
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hundred  pages.  For  an  ERP,  someone  needs  to  detennine  all  of  the  processes  that  the 
enterprise  does  and  figure  out  how  to  have  the  ERP  interface  with  them.  This  is  a  big  part 
of  the  Si’s  work,  and  specifying  the  requirements,  unless  it  is  done  at  a  very  high  level,  is 
difficult  from  the  outset.  Only  after  blueprinting  is  it  clear  which  processes  need  to  be 
designed  and  mapped  to  the  software;  even  then,  which  will  prove  difficult  and  which 
easy  may  not  be  known.  This  process  is  a  lot  more  like  research  and  development  than 
COTS  acquisition.  It  may  be  possible  to  do  a  cost  analysis  from  a  very  high  level  with 
cost-estimating  relationships  before  the  blueprinting  is  done,  but  to  our  knowledge  that 
type  of  analysis  has  not  been  done  for  any  DoD  ERP  system. 

3.  Consequences  of  Treating  an  ERP  as  COTS 

a.  Fixed  Price  Contracting 

In  the  late  1990s,  DoD  established  the  ESI.  Its  mission  is  to  lower  the  costs  of 
COTS  software  for  DoD,  the  Coast  Guard,  and  the  intelligence  community,  which  it  does 
by  negotiating  deals  with  COTS  providers.  In  2005,  when  ECSS  went  through  MS  A, 
ESI  was  offering  a  fixed  price  contract  vehicle  for  SI  services  that  could  be  competed 
among  five  companies:  Accenture,  Bearing  Point,  CSC,  Deloitte,  and  IBM.'  Many  of 
the  ERPs  whose  problems  were  written  about  by  the  GAO  in  October  2010  (Report  No. 
GAO-1 1-53)  were  started  in  this  era. 

The  theory  was  that  SI  services  could  be  thought  of  as  COTS,  so  the  contracts  were 
fixed  price  based  upon  the  specific  request  for  proposal  (RFP)  generated  by  the  program. 
An  assumption  of  fixed  price  contracts  is  that  cost  is  fairly  precisely  known.  Not  only  is 
this  assumption  incorrect,  cost  growth  was  the  rule — not  the  exception — in  ERP  SI 
contracts.  As  the  ECSS  program  went  on,  the  PO  realized  that  requirements  needed  to  be 
added  or  changed  to  make  ECSS  useful,  because  the  RFP  was  insufficiently  precise.  Each 
change  resulted  in  a  negotiation.  Based  on  this  experience,  it  is  not  clear  that  anybody 
could  write  an  RFP  with  sufficient  precision  to  execute  with  no  changes  before  the 
blueprinting  is  complete.  In  cost  reimbursement  contracts,  such  changes  do  not  involve 
much  negotiation;  there  might  be  a  cost  estimate,  but,  ultimately,  the  government  would 
order  the  work  and  the  contractor  would  do  it  and  bill  the  cost.  In  fixed  price  contracts, 
however,  each  change  must  have  a  negotiated  price.  ECSS  also  has  used  some  time  and 
materials  arrangements  that  have  negotiated  rates  instead  of  price. 
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According  to  ESI’s  webpage  (http://www.esi.mil/contentview. asp x?id=133&type=l,  accessed  1  April 
2011),  these  SI  contracts  were  ended  as  of  13  July  2009.  The  webpage  reports  they  will  be  replaced,  but 
so  far  there  is  nothing  new.  We  do  not  think  this  contract  will  come  back,  because  SI  services  are  not 


COTS. 
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It  is  generally  accepted  that  fixed  price  and  time  and  materials  contracts  are  not  well 
suited  to  research  and  development;  while  ERP  implementation  is  not  exactly  research,  it 
is  similar  enough  that  this  contract  vehicle  has  caused  problems.  The  PM  of  ECSS  said 
the  contract  with  CSC  was  not  an  ideal  vehicle,  but  he  was  working  with  it.  In  some  other 
programs  the  contract  was  changed  or  replaced.  One  problem  for  cost  analysts  is  that 
because  ERPs  have  been  bought  with  fixed  price  contracts,  there  is  no  cost  reporting  to 
improve  cost  estimates. 

b.  COTS  Acquisition  and  Cost  Reporting 

In  studying  DoD  programs,  it  is  common  for  analysts  to  use  contract  perfonnance 
reports  (CPRs)  and  contractor  cost  data  reports  (CCDRs)  to  track  the  cost  of  work.  For 
ECSS,  these  do  not  exist.  The  ECSS  PM  and  some  other  ERP  experts  argued  that  for  an 
ERP  it  does  not  make  sense  to  track  progress  relative  to  a  work  breakdown  structure,  as 
these  reports  require.  Although  we  do  not  have  a  position  with  regard  to  that  statement, 
the  fact  remains  that  there  is  currently  almost  no  cost  reporting  in  DoD  ERP  acquisitions, 
and  there  probably  will  not  be  any  cost  reporting  unless  the  regulations  governing  these 
acquisitions  are  overhauled.  The  repercussions  of  such  rule  changes  may  be  significant 
and  should  be  analyzed  before  they  are  adopted. 

Major  defense  contractors  are  accustomed  to  working  with  the  government  and 
reporting  their  costs.  When  these  firms  sign  cost  reimbursement  type  contracts  with  the 
government,  they  already  have  accountants  and  systems  that  are  fully  prepared  to  report 
their  costs  in  a  way  that  satisfies  the  government  customer  and  brings  in  profit.  They 
understand  reimbursable  costs  and  how  to  structure  their  business. 

CSC,  Oracle,  and  most  firms  in  the  ERP  field  do  not  use  cost-type  contracts.  In  their 
commercial  work,  most  Sis  use  a  combination  of  fixed  price  and  time  &  materials  type 
contracts,  while  COTS  suppliers  charge  licensing  fees.  They  are  not  set  up  to  share  their 
cost  data  as  major  defense  contractors  are.  If  cost  reporting  were  required,  some 
companies  might  build  the  expertise  to  do  it,  but  other  suppliers  might  decide  that  federal 
government  contracts  are  not  worth  the  trouble,  and  exit  the  market.  Cost  reporting 
expertise  is  also  expensive,  so  the  COTS  suppliers  would  have  higher  costs.  Some  of 
them  might  instead  team  up  with  traditional  defense  contractors  so  they  could  work  as 
subcontractors  instead  of  primes.  This,  too,  could  increase  the  cost  of  the  programs. 

B.  CSC’s  Performance 

The  CSC  employees  who  are  working  on  ECSS  are  located  near  the  PO  government 
staff  and  they  share  infonnation  informally.  Without  standard  cost  reporting  and  CPRs, 
this  informal  communication  may  be  the  PM’s  best  insight  into  CSC’s  performance.  We 
also  obtained  a  number  of  monthly  and  weekly  briefings — called  program  monthly 
reports  (PMR)  and  program  interim  reports  (PIR) — that  are  generated  by  CSC.  Although 
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these  reports  do  not  have  any  consistent  cost  numbers  in  them,  they  do  sometimes  track 
contract  line  items. 

Each  PMR  from  14  April  2010  through  14  December  2010  contained  two  lists  of 
tasks  related  to  each  pilot  of  Rl:  tasks  to  complete  in  the  next  month  and  tasks  completed 
in  the  previous  month.  Using  these  lists,  we  tracked  whether  CSC  was  accomplishing 
what  it  had  set  out  to  accomplish  in  the  previous  month.  Table  1  shows  what  we  found. 


Table  1.  Analysis  of  Tasks  Reported  in  CSC’s  PMRs 


Work  Area 

Tasks 

Completed 

As  Planned 

Tasks 

Completed 

Late 

Tasks  With 
Inconclusive 
Status 

Release  1  Pilot  A  (Rl  PA) 

16  (52%) 

2  (6%) 

13  (42%) 

Rl  PB 

23  (52%) 

7  (16%) 

14  (32%) 

Rl  PC 

23  (37%) 

5  (8%) 

35  (56%) 

Total 

62  (45%) 

14(10%) 

62  (45%) 

Forty-five  percent  of  all  tasks  with  specific  end  dates  were  reported  as  completed  on 
time.  Ten  percent  of  tasks  were  reported  to  be  completed  a  month  or  more  after  they  were 
due.  Forty-five  percent  of  tasks  were  never  reported  as  completed.  We  do  not  know  if  the 
tasks  in  this  last  group  were  completed  on  time,  completed  late,  still  ongoing,  or 
discarded  as  no  longer  necessary. 

We  used  only  PMRs  to  find  tasks  to  be  completed,  but  both  PMRs  and  PIRs  were 
used  to  verify  task  completion.  The  last  PMR  from  December  2010  was  only  used  to 
check  on  the  completion  of  previous  tasks,  while  all  other  PMRs  were  also  used  to  see 
what  tasks  were  scheduled. 

Because  we  do  not  know  the  size  or  importance  of  any  particular  task,  it  is  not  fair 
to  assume  that  all  tasks  are  equally  important  to  the  success  of  ECSS;  it  may  be  that  the 
most  important  tasks  are  in  the  45  percent  that  were  completed  on  time.  This  makes  us 
unsure  as  to  whether  CSC  is  doing  a  good  job  with  regard  to  such  measures  as  cost, 
schedule,  test  discrepancy  reports,  etc.  If  they  are  not,  it  could  contribute  to  any  part  of 
the  cost  growth. 

C.  ECSS  Is  Big 

The  size  of  ECSS  is  relevant.  The  only  DoD  ERPs  that  come  close  to  ECSS  in  size 
are  Global  Combat  Support  System  Army  (GCSS-Army)  and  Navy  ERP.  Several  metrics 
for  the  size  of  these  ERPs  are  presented  in  Table  2.  Finding  comparable  data  for  the  size 
of  commercial  ERPs  is  not  easy,  but  there  are  few  companies  with  as  many  employees  as 
ECSS  is  projected  to  have  users,  and  the  Air  Force  logistics  enterprise  is  more 
complicated  and  diverse  than  most  companies.  Cynthia  Rettig  wrote  a  paper  critical  of 
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ERPs  in  the  Fall  2007  MIT  Sloan  Management  Review  that  claims  that  the  average 
installation  cost  of  an  ERP  is  $  1 5  million,  but  large  organizations  often  spend  hundreds 
of  millions  of  dollars.  Even  the  smallest  of  the  DoD  ERPs  written  about  in  the  2010  GAO 
report  were  expected  to  cost  over  $100  million  at  MS  A,  and  DoD’s  largest,  including 
ECSS,  are  an  order  of  magnitude  larger  than  that. 


Table  2.  ERP  Size  Metrics  from  GAO-1 1-53  for  the  Three  Most  Expensive  ERPs 


Current  Life 
Cycle  Cost 
Estimate 

Legacy 

Systems 

Replaced 

Number  of 
System 
Interfaces 

Number  of 
Users 

Number  of 
Locations 

ECSS 

$5.2B 

240 

830 

250,000 

186 

GCSS-Army 

$3.9B 

7 

106 

169,880 

379 

Navy  ERP 

$2.4B 

98 

51 

66,000 

53 

Source:  All  Data  is  from  GAO-1 1-53  and  was  current  on  31  December  2009. 

Note  that  as  of  today,  ECSS  R1  is  expected  to  have  40k  users  and  157  system  interfaces;  the  numbers  in 
the  table  are  for  all  of  ECSS. 


According  to  GAO-11-53,  ECSS  is  the  biggest  in  all  metrics  except  for  number  of 
locations,  where  it  is  second.  The  life  cycle  cost  estimate  of  $5.2  billion  is  the  most 
expensive  in  DoD.  The  large  scope  of  ECSS  makes  the  research-like  portion  of  the 
program  more  complicated.  Even  if  cost  does  scale  linearly  with  regard  to  legacy  systems 
replaced  and  with  the  number  of  system  interfaces,  there  are  likely  to  be  interaction  terms 
that  make  the  costs  more  difficult  to  project.  In  other  words,  the  cost  of  adding  one  more 
system  to  replace — holding  everything  else  constant — may  be  straightforward  to 
calculate,  but  when  one  system  is  added  and  one  more  interface  is  added  the  cost  goes  up 
even  more  than  the  sum  of  the  system  and  the  interface  because  the  newly  configured 
system  must  interact  with  all  the  old  interfaces  plus  the  new  one. 

The  large  size  of  ECSS  made  it  more  difficult  to  estimate  the  cost.  This  would  have 
contributed  to  the  risk  estimate  as  well  as  the  cost  growth  in  every  other  category. 

D.  The  “Fourth  Estate” 

DoD  Components  are  expected  or  required  to  use  a  wide  variety  of  IT  systems,  and 
this  makes  them  more  complicated  than  commercial  ERPs.  For  example,  the  Defense 
Contract  Management  Agency  has  a  system  called  Wide  Area  Workflow  (WAWF)  to 
handle  electronic  invoicing.  The  Oracle  COTS  software  has  an  intrinsic  capability  to  do 
invoicing,  but  it  cannot  be  used;  the  ERP  must  work  with  the  existing  system  instead. 
Since  the  Air  Force  cannot  control  how  WAWF  works,  it  has  to  generate  RICE  objects  to 


Cynthia  Rettig,  “The  Trouble  with  Enterprise  Software,”  MIT  Sloan  Management  Review  49,  No.  1 
(Fall  2007). 


24 


work  with  it.  In  a  sense,  this  breaks  the  continuity  of  the  ERP,  because  ECSS  does  not 
control  a  process  from  end-to-end  as  the  COTS  developer  intended.  This  makes  its 
operation  more  complicated  even  though  it  is  doing  less.  Also,  if  WAWF  changes  its 
interface,  which  is  outside  the  control  of  the  ECSS  program,  then  ECSS  must  spend  its 
own  money  to  adapt  to  it. 

Figure  6  shows  ECSS’s  content  in  a  chart  from  the  PO.  Each  of  the  purple  boxes  on 
the  top  represents  an  organization  outside  of  the  Air  Force  with  which  ECSS  needs  to 
interact.  Many  of  these  organizations  have  multiple  systems  that  must  communicate  with 
ECSS. 
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Figure  6.  Program  Content/Deliverables  from  ECSS  PM  5  January  2011  Briefing 


It  is  probably  the  case  that  one  such  system  would  not  present  a  terrible  hurdle  for 
an  ERP,  but  there  are  many  DoD  systems  that  control  aspects  of  what  an  ERP  would 
normally  do.  This  problem  is  sometimes  called  the  “fourth  estate”  problem  by  ERP 
experts  in  and  around  DoD,  including  within  the  ECSS  PO  and  the  Business 
Transfonnation  Agency  (BTA).  It  is  called  the  fourth  estate  because  it  is  a  collection  of 
outside  forces  that  shape  the  way  the  ERP  works,  but  are  not  part  of  it. 

BTA  personnel  told  us  that  in  the  commercial  world,  an  ERP  typically  would 
replace  most  of  the  other  systems  with  which  DoD  ERPs  need  to  communicate.  An  ERP 
with  fewer  external  processes  is  easier  to  implement  because  it  needs  fewer  RICE  objects 
and  has  more  continuity  in  the  data.  This  is  a  general  difficulty  of  all  DoD  ERPs,  and, 
according  to  BTA,  it  is  a  serious  one. 

If  the  fourth  estate — which  existed  from  the  start — had  been  well  understood  from 
the  beginning,  it  would  have  increased  the  proposed  cost  of  ECSS,  but  would  not  have 
created  cost  growth  during  the  program’s  execution.  The  only  category  of  growth  from 
the  PM’s  list  that  this  would  have  contributed  to  is  risk. 

E.  Management  Authority  and  Decision  Velocity 

According  to  Rettig  and  other  ERP  literature,  a  commonly  reported  problem  in  ERP 
implementation  is  lack  of  authority  at  the  top.  In  order  for  an  ERP  implementation  to 
succeed,  it  must  be  adopted  by  many  people  throughout  the  enterprise.  The  ERP  must 
conform  to  their  needs,  but  people  also  need  to  change  the  way  they  work  to  take 
advantage  of  the  ERP.  If  the  leader  of  the  ERP  effort  has  insufficient  authority  over  the 
users,  there  will  be  problems. 

While  working  on  the  fit-gap  analysis,  disagreements  will  occur  over  how  the  ERP 
should  be  configured.  In  the  simplest  case,  the  people  who  currently  run  a  process  will 
send  a  representative  to  meet  with  the  SI  personnel  to  discuss  how  a  process  should  be 
standardized.  The  SI  will  present  the  COTS  processes,  and  the  practitioner  may  contend 
that  none  are  suitable  as  is.  Coming  to  agreement  is  often  difficult  and  the  rate  at  which 
decisions  are  made  is  referred  to  as  “decision  velocity.”  This  problem  can  get 
significantly  worse  if  there  are  multiple  groups  that  currently  do  the  same  process — for 
example,  at  different  bases  or  from  different  communities — in  different  ways,  because  all 
must  be  satisfied  with  the  final  decision.  These  meetings  are  a  significant  portion  of  the 
cost  of  configuring  an  ERP.  Once  a  conclusion  is  reached  and  the  ERP  is  implemented, 
there  is  still  the  difficulty  of  getting  the  various  users  to  adapt  to  the  new  ways  of  doing 
business. 

Umble,  Haft,  and  Umble  wrote  in  the  European  Journal  of  Operation  Research  that 
a  successful  ERP  implementation  requires  “strong  leadership  commitment  and 


27 


22 

participation  by  top  management.”  '  The  company’s  president  needs  to  be  associated 
with  the  project  to  succeed  because  this  sort  of  program  goes  far  beyond  IT.  If  the  top 
person  pushing  for  ECSS  were  the  Air  Force  Chief  of  Staff,  issues  would  be  easier  to 
resolve  and  decision  velocity  would  be  higher.'  On  the  other  hand,  if  the  top  person 
were  a  field  grade  officer,  the  experts  from  around  the  Air  Force  would  feel  no 
compulsion  to  yield  to  the  SI — the  result  being  either  an  ERP  with  a  great  deal  of  RICE 
objects  increasing  the  cost  to  build  and  maintain  it  and  minimal  unifonnity  from  one 
group  to  another,  or  exceedingly  slow  decision  velocity.  The  situation  at  ECSS  is 
between  these  two.  Part  of  the  reason  a  brigadier  general  was  selected  to  run  ECSS  and 
be  the  PEO  for  all  logistics  IT  systems  was  to  provide  the  authority  required  to  push 
development  forward. 

If  insufficient  management  authority  has  caused  cost  growth  in  this  program,  it 
would  be  part  of  the  risk,  schedule  delay,  and  deliberate  delivery  strategy  categories 
previously  discussed. 

F.  ERPs  in  Business  Today 

Another  root  cause  may  be  that  ERPs  are,  in  general,  not  good  investments.  It  is  our 
impression  from  the  literature,  like  Rettig  and  Go  sain,  that  ERPs  have  a  reputation  for 
trouble  that  includes  cost  overruns  and  schedule  slips  in  the  implementation  phase  and 
disappointment  with  the  systems  once  they  are  running. 

The  October  2010  GAO  report  makes  it  clear  that  ERPs  are  difficult  to  implement  in 
DoD.  There  is  also  literature  that  says  implementing  commercial  ERPs  is  difficult.  In 
2007,  Rettig  wrote  about  the  difficulties  of  ERPs  and  cited  the  attention-grabbing  statistic 
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that  75  percent  of  ERP  implementations  were  considered  failures. 


Elizabeth  J.  Umble,  Ronald  R.  Haft,  and  M.  Michael  Umble,  “Enterprise  Resource  Planning: 
Implementation  Procedures  and  Critical  Success  Factors,”  European  Journal  of  Operational  Research 
146  (2003):  241-257. 

Even  the  Air  Force  Chief  of  Staff  may  not  have  sufficient  pull  to  make  the  Air  Force  adopt  an  ERP  like 
ECSS  as  quickly  as  some  might  like,  but  nobody  else  could  do  it  faster. 

Sanjay  Gosain,  “Enterprise  Information  Systems  as  Objects  and  Carriers  of  Institutional  Forces:  The 
New  Iron  Cage?”  Journal  of  the  Association  for  Information  Systems  5,  No  4,  Article  6  (2004). 

Tracking  down  this  statistic  to  its  source  showed  that  it  came  from  a  survey-based  analysis  in  which  34 
of  50  surveyed  companies  responded.  There  are  reasons  to  think  that  the  survey  might  have  been  biased 
toward  successful  ERPs,  as  the  50  were  selected  by  ERP  COTS  software  vendors.  For  a  full  discussion, 
see  K.  K.  Hong  and  Y.  G.  Kim,  “The  Critical  Success  Factors  for  ERP  Implementation:  An 
Organizational  Fit  Perspective,”  Information  &  Management  40,  No.  1  (October  2002):  25. 
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G.  Root  Cause  Summary 

One  question  asked  in  every  root  cause  analysis  for  D,PARCA  is  whether  this 
problem  was  caused  before  the  program’s  inception  or  after.  Since  “birth”  is  generally 
defined  as  MS  B,  this  program  has  not  yet  been  born.  It  is  conceivable  that  ECSS  R1  will 
receive  MS  B  authority  sometime  soon,  with  a  baseline  it  can  stick  to,  and  never  suffer  a 
cost  breach. 

None  of  the  root  causes  of  the  cost  growth  in  the  ECSS  program  came  from 
unpredictable  exogenous  events  since  MS  A.  The  deepest  root  cause  of  cost  growth  and 
schedule  delay  in  ECSS  came  about  because  the  people  who  began  this  program  had  an 
insufficient  understanding  of  what  it  entailed — its  size,  its  scope,  and  the  nature  of  the 
problems  presented.  As  the  scope  of  ECSS  is  so  big,  it  is  possible  that  nobody  in  the 
world  really  knew  how  long  it  would  take  or  what  it  would  cost  to  acquire  this  system.  It 
may  also  be  that  it  is  impossible  to  know  how  much  a  large  ERP  will  cost  until  the 
blueprinting  is  done,  which  is  a  significant  portion  of  the  cost. 

Because  ECSS  is  still  pre-MS  B,  it  is  possible  that  the  cost  growth  and  schedule 
delay  to  date  can  be  dismissed  as  growing  pains  and  this  program  will  settle  down  into  a 
good  performer  for  the  Air  Force.  We  are  skeptical,  though,  because  the  data  about 
CSC’s  performance  problems  are  from  early  FY  2011  and  the  PM  wants  his  R1  APB 
soon. 

Of  the  standard  categories  for  cost  growth  that  the  Weapons  System  Acquisition 
Refonn  Act  of  2009  lays  out,  this  program  clearly  suffered  from  one  “unanticipated 
design,  engineering,  manufacturing,  or  technology  integration  issue[s]  arising  during 
program  performance”  (the  data  importation  issues).  There  is  also  evidence  that  can  be 
interpreted  as  “poor  perfonnance  by  government  or  contractor  personnel  responsible  for 
program  management”  in  that  they  did  not  fully  report  shortfalls  in  program 
accomplishments.  Lastly,  the  program  was  launched  with  a  grossly  insufficient 
understanding  of  the  problems  involved  and,  consequently,  had  an  unrealistically  low 
cost  estimate. 
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Appendix  A. 

Hidden  Costs  Typically  Overlooked 


There  are  several  components  to  every  successful  ERP  implementation  that  are 
often  overlooked  because  they  are  external  to  the  primary  effort  of  bringing  the  new 
system  online.  These  additional  tasks  can  add  unexpected  costs  and  effort  to  the  process, 
yet  an  ERP  implementation  will  fail  without  them.  Some  of  this  work  must  be  conducted 
within  the  organization,  while  other  work  requires  outside  vendors  to  go  beyond  their 
basic  ERP  development  tasks.  We  call  them  out  here  because,  while  they  are  mentioned 
above,  they  do  not  contribute  to  the  official  costs  of  the  program  that  are  recorded  in 
budget  requests  or  PO  reporting.  Rather  they  will  come  out  of  the  activities  of  other 
commands  generally  in  the  fonn  of  overtime  hours  worked  or  reduced  productivity. 

First,  the  organization  must  engage  personnel  at  every  level  from  the  onset.  Users  of 
the  system  and  of  its  reports  must  meet  to  work  through  the  suitability  of  the  planned 
automation  and  its  constraints.  Leadership  must  consider  the  results  of  these  meetings  to 
decide  if  ERP  is  appropriate  for  the  organization  as  well  as  the  depth  and  breadth  of  its 
implementation. 

Second,  the  business  processes  must  be  documented  and  examined  for  suitability  of 
automation.  Some  procedures  may  have  to  be  modified,  some  replaced,  and  some  left 
external  to  the  automation  (e.g.,  improvement  analysis).  Many  organizations  choose  to 
streamline  their  processes  prior  to  selecting  an  ERP  product  or  vendor  while  others  prefer 
to  select  the  software  first  (perhaps  for  compatibility  across  a  larger  organization)  and 
later  fit  their  processes  to  the  constraints  of  the  software.  The  optimum  approach  may  be 
a  combination  of  high-level  re-engineering  to  accommodate  known  deficiencies  or 
inconsistencies  in  current  business  processes  followed  by  a  fit-gap  analysis  to  select  the 
software  that  can  best  accommodate  the  organization’s  needs.  After  software  is  selected 
and  its  limitations  are  known,  progress  can  be  made  at  a  more  granular  level  to  adapt 
process  details  as  needed,  as  well  as  to  consider  customizations  to  the  software.  An  ERP 
vendor  or  SI  is  typically  hired  to  assist  with  these  steps. 

Finally,  personnel  who  will  use  the  ERP  or  its  reports  must  be  trained,  not  only  in 
the  re-engineered  processes  but  also  in  the  use  of  the  software  itself.  While  the  cost  of 
training  may  be  included  in  vendor  estimates,  the  hours  spent  by  personnel  to  attend 
training,  in  addition  to  the  loss  in  productivity  during  the  conversion  period  when  new 
and  old  processes  often  overlap,  are  frequently  external  to  the  cost  estimates  for  standing 
up  the  new  system. 
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Appendix  B. 

ECSS  and  IDA’s  Conditions  for  Successful  ERP 

Implementations 


In  2011,  IDA  produced  a  paper  on  ERP  programs  for  the  OSD  Comptroller  and 
the  Deputy  Chief  Management  Officer,  which  was  required  by  the  House  Anned 
Services  Committee. 1  The  IDA  analysis  resulted  in  a  list  of  eight  necessary  conditions 
for  a  successful  ERP  implementation,  meaning  that  “benefits  and  operational 
improvements  are  realized  to  the  planned  extent.”  The  idea  behind  these  conditions  is 
that  if  they  are  not  all  present,  the  attempt  will  either  exceed  the  expected  costs  or  fall 
short  on  benefits.  By  our  analysis,  the  Air  Force  ECSS  program  currently  meets  only 
one  of  these  necessary  conditions  and  is  actively  working  towards  meeting  another, 
although  they  are  not  there  yet.  We  cannot  judge  the  status  of  two  others.  This  appendix 
contains  our  assessment  of  ECSS  relative  to  each  condition. 


#  Condition  Assessment 

1  Sustained  involvement  of  senior  leadership  with  authority  over  and  Fail 

accountability  for  the  definition  and  execution  of  all  end-to-end 
processes  impacted  by  the  ERP. 


The  Air  Force  has  come  to  understand  this  problem  and  is  attempting  to  address  it; 
however,  there  is  a  long  way  to  go  before  this  condition  is  met.  The  PM  has  a  great  deal 
of  control  within  the  Air  Force  and  it  may  be  enough,  but  he  has  no  control  over  other 
DoD  systems  that  are  vitally  important  to  ECSS  functioning. 

The  Air  Force  addressed  the  internal  issues  in  July  2009  by  appointing  a  brigadier 
general  to  the  role  of  program  manager  for  ECSS  as  well  as  PEO  for  all  logistics 
systems  that  ultimately  will  be  replaced  by  ECSS.  The  PEO/PM  does  have  control  over 
much  of  the  Air  Force’s  logistics-related  IT  systems. 

Neither  the  PEO  nor  the  Air  Force  controls  how  other  parts  of  DoD  operate  and 
handle  data.  This  “fourth  estate”  has  caused  problems  for  other  ERPs  and  will  cause 
problems  for  ECSS.  Reducing  the  fourth  estate’s  impact  would  require  granting  the 


Paul  K.  Ketrick,  John  W.  Bailey,  Marilee  O.  Cunningham,  Laura  A.  Odell,  Graeme  R.  Douglas, 
Dawn  M.  Floyd,  and  Anthony  Insolia,  “Assessment  of  DoD  Enterprise  Resource  Planning  Business 
Systems,”  IDA  Paper  P-4691  (Draft  Final),  Institute  for  Defense  Analyses,  February  2011. 
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enterprises  that  are  implementing  ERPs  more  autonomy  in  their  business  practices  than 
currently  exists. 


#  Condition  Assessment 

2  Leadership  willingness  and  ability  to  make  hard  decisions  relative  to  Unclear 

proceeding  or  not  proceeding  with  an  implementation  based  on 
program  performance. 


This  program  has  a  program  manager,  a  sponsor,  and  a  milestone  decision 
authority.  In  addition,  higher  level  personnel,  such  as  the  Secretary  of  the  Air  Force,  the 
Secretary  of  Defense,  and  even  some  officials  in  the  White  House,  are  aware  of  this 
program.  It  is  unclear  who  would  have  the  authority  to  cancel  it  if  they  thought  it  was 
warranted. 


#  Condition  Assessment 

3  Strong  integrated  governance  that  includes  representation  of  and  Fail 

participation  by  all  impacted  stakeholders.  The  representatives 
must  have  the  authority  to  make  decisions  that  are  binding  on  the 
communities  they  represent.  Decisions  must  be  made  rapidly  and 
the  effectiveness  of  the  governance  must  be  actively  measured  and 
reported. 


As  with  condition  #1,  the  PEO  has  sufficient  authority  within  the  Air  Force,  but 
ECSS  has  many  ties  outside  the  Service  over  which  the  PEO  and  his  superiors  have 
little  or  no  control.  To  improve  this  condition,  the  Air  Force  logistics  enterprise  would 
need  more  autonomy  over  its  business  practices  than  it  currently  has,  although  granting 
that  authority  to  the  Air  Force  would  create  new  issues  for  study. 


#  Condition  Assessment 

4  An  organizational  operating  model  (structure  and  process)  aligned  Pass 

to  the  design  of  the  ERP  with  minimal  requirements  to  cross- 
organizational  boundaries  and  which  execute  components  of  a 
process  outside  of  the  ERP,  thus  breaking  the  inherent  integration 
of  the  ERP. 


The  PM’s  team  is  adamant  about  fighting  vested  interests  within  the  Air  Force  to 
get  them  to  change  their  processes  to  work  with  the  Oracle  software  as  much  as 
possible. 
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#  Condition  Assessment 

5  A  strategy  and  approach  that  address  the  root  cause  (not  just  the  Unclear 

symptoms)  of  the  problems  being  solved  and  the  measurable 
operational  improvement  to  be  gained  by  solving  them. 


This  condition  is  bigger  than  ECSS.  An  ERP  cannot  be  successful  by  itself 
because  it  is  merely  a  tool.  This  condition  asks  if  the  ERP  is  part  of  a  broader  strategy 
that  will  improve  the  enterprise.  This  broader  strategy  comes  from  the  Air  Staff  and,  in 
particular,  the  sponsor’s  office.  It  is  called  Expeditionary  Logistics  for  the  21st  Century 
or  eLog21.  It  was  beyond  the  scope  of  this  study  to  detennine  whether  or  not  this  plan 
attacks  the  deepest  issues  in  Air  Force  logistics. 


#  Condition  Assessment 

6  Personnel  with  the  requisite  skill  set  and  experience  necessary  to  Fail  -  probably 
define  and  execute  an  ERP  implementation  (e.g.,  source  selection,  improving 

contracting,  vendor  management,  change  management,  technical 
oversight). 


In  the  early  days  of  this  program,  the  plan  was  to  buy  Oracle’s  ERP  software  and 
integrate  it  with  two  other  software  packages:  IFS  for  maintenance  and  Click 
Commerce  for  supply  planning.  Nobody  had  experience  integrating  these  products 
because  it  had  never  been  done,  although  the  Air  Force  treated  this  as  a  COTS 
acquisition.  This  anecdote,  along  with  other  evidence  discussed  in  the  main  body  of  this 
paper,  suggests  that  the  Air  Force  did  not  fully  understand  what  they  were  buying.  The 
Air  Force  has  since  decided  that  Oracle’s  latest  products  can  give  them  the  capability 
they  need.  However,  the  Air  Force  has  not  demonstrated  that  it  has  acquired  more 
expertise  on  the  implementation  of  ERPs.  We  marked  this  condition  as  “probably 
improving”  because  the  ECSS  PO  does  contain  numerous  highly  capable  people  who 
have  been  working  with  CSC  for  several  years. 


#  Condition  Assessment 

7  Defined  metrics  for  operational  improvement  to  be  gained,  Fail 

supported  by  a  baseline  describing  existing  business  performance. 


The  Air  Force  created  a  105-page  Capability  Document  and  a  66-page  TEMP, 
both  of  which  came  into  their  current  form  in  Spring  2010.  The  TEMP  has  63  MOEs 
and  Measures  of  Suitability  (MOSs)  that  R1  is  supposed  to  meet.  Nine  of  these  relate  to 
the  performance  of  the  Air  Force.  Thirty-nine  describe  the  capabilities  of  the 
program — measures  that  should  help  to  improve  the  Air  Force,  but  are  not  direct 
measures  of  how  well  the  Air  Force  is  doing.  Thirteen  are  about  the  capability  of  the 
computer  system,  which  should  make  the  program  better,  and  a  better  program,  in  turn, 
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should  make  the  Air  Force  better — although  that  is  not  guaranteed.  The  last  two  relate 
to  satisfaction  of  guidance  and  standards,  which  may  be  wise  but  are  not  measures  of 
improvements  to  the  Air  Force. 

Of  the  nine  MOEs  that  relate  to  the  Air  Force,  five  of  them  have  criteria  listed  as 
“TBD,”  so  the  requirement  has  not  yet  been  set.  The  other  four  are  shown  in  Table  B-l. 


Table  B-1.  Requirements  in  the  ECSS  TEMP  for  Air  Force  Performance 


Test  Descriptor 

Requirement 

Measure 

Criteria 

Operations  and 
sustainment  cost 
savings 

ECSS  shall  be  less 
expensive  to  operate  and 
sustain  (O&S)  than  the 
subsumed  systems  in 
constant  FY  2005  dollars 

(Key  System 

Attribute)  MOS  3.19: 
Operations  and 
sustainment  cost 
savings 

Programmed  ECSS, 
release  1,  O&S  dollars: 
<99%  of  FY  2005 
dollars  expended  for 
planned  subsumed 
systems 

Percent  auditable 
transactions 

ECSS  shall  enable 
auditable  transactions 
involving  vehicles, 
equipment,  mobility  gear, 
and  tools 

MOE  2.5:  Percent  of 

auditable 

transactions 

>  98% 

Percent  of 
successful  on- 
time  deliveries 

(Key  Performance 
Parameter  (KPP))  ECSS 
shall  enable  on-time 
delivery 

(KPP)  MOE  1.2: 
Delivery  Rate.  Not  a 
measure  exclusive 
to  ECSS 

>  75% 

Percent  of  orders 
filled 

(KPP)  ECSS  shall 
enable  full  order 

(KPP)  MOE  1.3:  Fill 
Rate.  Not  a  measure 
exclusive  to  ECSS 

>  95% 

These  four  requirements  seem  fairly  powerful,  but  only  MOS  3.19  really  focuses 
on  the  capability  of  the  Air  Force — by  spending  less  money  on  logistics,  there  should 
be  more  money  available  elsewhere.  The  Air  Force  would  like  to  have  more  orders 
filled  and  delivered  on  time,  but  if,  for  example,  the  system  only  increases  on-time 
delivery  by  lengthening  schedules  and  maintenance  happens  at  the  same  speed,  even 
that  is  not  a  significant  improvement,  and  it  may  even  make  the  Air  Force  worse  off.  ft 
is  possible  that  all  of  these  metrics  could  be  satisfied — possibly  going  well  above  the 
baseline — and  the  Air  Force  would  still  be  less  well  off  for  having  implemented  ECSS. 
Suppose,  for  example,  that  ECSS  meets  all  four  requirements  above  but  the  availability 
rate  for  some  aircraft  types  falls  because  the  system  forces  mechanics  for  those  types  to 
work  less  efficiently  than  they  had  before.  What  is  needed  are  MOEs  saying  that  the 
sortie  rates  or  availability  rates  of  all  aircraft  in  the  inventory  must  not  decrease — 
possibly  with  a  nonnalization  for  aircraft  aging. 

Also,  it  is  not  clear  that  98  percent  of  auditable  transactions  is  good  enough  to  get 
the  Air  Force  the  “clean  audit”  that  the  Congress  requires. 
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Finally,  there  does  not  seem  to  be  any  document  containing  baselines.  This  may 
be  in  part  because  there  are  as  many  different  baselines  as  there  are  many  different 
ways  tasks  are  performed  throughout  the  Air  Force. 


# 

Condition 

Assessment 

8 

Accurate,  consistent,  and  authoritative  data. 

Fail  -  but  being 
worked  on 

Air  Force  logistics  data  is  stored  on  many  different  systems  in  different  ways  all 
across  the  Air  Force.  This  problem  is  understood,  and  a  significant  portion  of  what  the 
ECSS  program  is  doing  is  trying  to  fix  the  various  datasets  to  make  them  accurate, 
consistent,  and  authoritative.  Currently  that  work  is  ongoing. 
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Appendix  C. 

Benefits  by  Year  and  Benefits  by  Release 


Table  C-1.  Benefits  by  Years  FY  2011-FY  2019  (millions  of  TY  dollars) 


Benefit  Title 

2011 

2012 

2013 

2014 

2015 

2016 

2017 

2018 

2019 

Fewer  Spares  /  Equipment  Needed  (MSD 
spend) 

5.47 

65.66 

257.17 

525.30 

738.73 

820.83 

Inventory  Holding  Costs 

1.10 

13.20 

51.70 

105.59 

148.50 

165.00 

Fewer  Spares  /  Equipment  Moved 

0.01 

0.09 

0.34 

0.69 

0.96 

1.07 

Better  Financial  Management 

0.00 

0.02 

0.04 

0.06 

0.07 

0.07 

Less  Man-Hours 

0.26 

0.79 

3.67 

8.65 

13.44 

15.86 

Supply  Chain  Sub  Total 

0.00 

0.00 

0.00 

6.84 

79.76 

312.92 

640.29 

901.70 

1002.83 

Legacy  IT  -  Compliance  Modernization 

50.00 

140.00 

140.00 

140.00 

50.00 

0.00 

0.00 

0.00 

0.00 

Legacy  IT  Sustainment 

0.00 

0.00 

0.00 

33.52 

76.35 

182.93 

267.59 

307.12 

315.08 

Legacy  IT  Sub  Total 

50.00 

140.00 

140.00 

173.52 

126.35 

182.93 

267.59 

307.12 

315.08 

Total 

50.00 

140.00 

140.00 

180.36 

206.11 

495.85 

907.88 

1208.82 

1317.91 

Cumulative  Total 

50.00 

190.00 

330.00 

510.36 

716.47 

1212.32 

2120.20 

3329.02 

4646.93 
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Table  C-2.  Benefits  by  Years  FY  2020-FY  2026  (millions  of  TY  dollars) 


Benefit  Title 

2020 

2021 

2022 

2023 

2024 

2025 

2026 

Total 

Fewer  Spares  /  Equipment  Needed  (MSD 
spend) 

820.83 

820.83 

820.83 

820.83 

804.42 

640.25 

246.25 

7387.40 

Inventory  Holding  Costs 

165.00 

165.00 

165.00 

165.00 

161.70 

128.70 

49.50 

1484.99 

Fewer  Spares  /  Equipment  Moved 

1.07 

1.07 

1.07 

1.07 

1.05 

0.81 

0.32 

9.62 

Better  Financial  Management 

0.07 

0.07 

0.07 

0.07 

0.06 

0.01 

0.00 

0.61 

Less  Man-Hours 

15.86 

15.86 

15.86 

15.86 

15.07 

14.28 

7.14 

142.60 

Supply  Chain  Sub  Total 

1002.83 

1002.83 

1002.83 

1002.83 

982.30 

784.05 

303.21 

9025.22 

Legacy  IT  -  Compliance  Modernization 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

520.00 

Legacy  IT  Sustainment 

325.04 

337.50 

353.06 

372.51 

343.21 

246.23 

134.01 

3294.15 

Legacy  IT  Sub  Total 

325.04 

337.50 

353.06 

372.51 

343.21 

246.23 

134.01 

3814.15 

Total 

1327.87 

1340.33 

1355.89 

1375.34 

1325.51 

1030.28 

437.22 

12839.37 

Cumulative  Total 

5974.80 

7315.13 

8671.02 

10046.36 

11371.87 

12402.15 

12839.37 

Table  C-3.  Benefits  by  Release  (millions  of  TY  dollars) 


O&M  Benefit 

R1 

R2 

Low 

Likely 

High 

Low 

Likely 

High 

Fewer  Spares  /  Equipment  Needed  (MSD  spend) 

84.43 

147.75 

168.86 

844.28 

1477.48 

1688.55 

Inventory  Holding  Costs 

17.82 

29.70 

59.40 

178.20 

297.00 

593.99 

Fewer  Spares  /  Equipment  Moved 

0.14 

0.19 

0.29 

1.49 

2.12 

3.14 

Better  Financial  Management 

0.03 

0.03 

0.29 

0.38 

0.45 

0.76 

Less  Man-Hours 

4.75 

7.13 

14.26 

4.75 

7.13 

14.26 

Supply  Chain  Sub  Total 

107.17 

184.80 

243.10 

1029.10 

1784.18 

2300.70 

Legacy  IT  -  Compliance  Modernization 

72.91 

72.91 

72.91 

149.03 

149.03 

149.03 

Legacy  IT  Sustainment 

419.41 

419.41 

419.41 

906.64 

906.64 

906.64 

Legacy  IT  Sub  Total 

492.32 

492.32 

492.32 

1055.67 

1055.67 

1055.67 

Total 

599.49 

677.12 

735.42 

2084.77 

2839.85 

3356.37 
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Table  C-4.  Benefits  by  Release  (millions  of  TY  dollars) 


R3 

R4 

Total 

O&M  Benefit 

Low 

Likely 

High 

Low 

Likely 

High 

Low 

Likely 

High 

Fewer  Spares  /  Equipment  Needed 
(MSD  spend) 

2026.25 

3545.96 

4052.53 

1266.42 

2216.23 

2532.83 

4221.38 

7387.42 

8442.77 

Inventory  Holding  Costs 

427.68 

712.79 

1425.58 

267.30 

445.50 

890.99 

891.00 

1484.99 

2969.96 

Fewer  Spares  /  Equipment  Moved 

3.12 

4.43 

6.57 

2.03 

2.89 

4.28 

6.78 

9.63 

14.28 

Better  Financial  Management 

0.08 

0.09 

0.15 

0.03 

0.03 

0.05 

0.52 

0.60 

1.25 

Less  Man-Hours 

42.79 

64.18 

128.36 

42.79 

64.18 

128.36 

95.08 

142.62 

285.24 

Supply  Chain  Sub  Total 

2499.92 

4327.45 

5613.19 

1578.57 

2728.83 

3556.51 

5214.76 

9025.26 

11713.50 

Legacy  IT  -  Compliance  Modernization 

149.03 

149.03 

149.03 

149.03 

149.03 

149.03 

520.00 

520.00 

520.00 

Legacy  IT  Sustainment 

956.83 

956.83 

956.83 

1011.28 

1011.28 

1011.28 

3294.16 

3294.16 

3294.16 

Legacy  IT  Sub  Total 

1105.86 

1105.86 

1105.86 

1160.31 

1160.31 

1160.31 

3814.16 

3814.16 

3814.16 

Total 

3605.78 

5433.31 

6719.05 

2738.88 

3889.14 

4716.82 

9028.92 

12839.42 

15527.66 
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enough  people  understood  what  they  were  trying  to  acquire.  The  result  of  this  was  an  acquisition  that  was  not  well  suited  to  buying 
an  ERP  or  estimating  its  schedule  or  cost. 
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